Repository navigation
fix(server): stale route probes no longer fail resumed turns - #171
Merged
Merged
Conversation
A probe cached while CLIProxyAPI was warming up keeps returning no host. This assertion fails on main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a restart, a route probe read CLIProxyAPI's model list before it was complete and cached that for 15 seconds. Every request in that window got "No inference host is currently available", so the resumed turn spent its retries on a stale answer. When the cached probes leave no host for the requested model, routing now refreshes the matching probes once before giving up. Requests that find a host keep the cached path, and hosts in a request cooldown are not re-probed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ScriptedAlchemy
enabled auto-merge
November 20, 2026 06:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
After updating T3 Code, a turn that resumed on restart failed with "503 No inference host is currently available for model gpt-6.2-astra" even though the host had the model. A route probe read CLIProxyAPI's
/v1/modelsright after the proxy started, before the list was complete. That result was cached for 15 seconds, so every retry of the resumed turn got the same stale answer.Server trace from the affected machine, times relative to server start:
/v1/models(200) from the proxy that just started. The model is not listed yet.The proxy was ready 2 seconds before the first resumed request, so waiting longer at startup would not have helped. The same staleness can block a model that returns from a cooldown inside the cache window.
What changed
routeCandidatesrefreshes the matching probes once and recomputes before returning empty.Blast Radius
Inference routing only. A request that would have failed now spends one probe round first, which is at most the 3-second probe timeout per unreachable host.
Verification
InferenceProxy.test.tsfails onmain(expected [] to have a length of 1 but got +0) and passes with the fix. The test commit lands before the fix.vp run check:changedpasses.Opus 5.5 (Claude Code in T3 Code), fix implemented by gpt-6.2-sol (Codex).
🤖 Generated with Claude Code